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11.2.7 Solved Problems 

Problem 1 

Consider the Markov chain with three states, S = {1, 2, 3}, that has the following transition 
matrix 


1 
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1 
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P = 



1 

2 


1 

2 
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a. Draw the state transition diagram for this chain. 

1 

b. If we know P(X l = 1) = P(X ] = 2) = find P(X l = 3,X 2 = 2,X 3 = 1). 
• Solution 

o a. The state transition diagram is shown in Figure 11.6 


I 

4 



Figure 11.6 - A state transition diagram. 


b. First, we obtain 
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P(X\ = 3) = 1 - P(X\ = 1) - P(X\ = 2) 


1 1 



1 

2 ' 


We can now write 


P(X j = 3,X 2 = 2,X 3 = 1 ) = P(Xj = 3) -p 32 -/> 21 

1 1 1 
" 2 ’ 2 ’ 3 
1 

~ 12 ' 


Problem 2 

Consider the Markov chain in Figure 11.17. There are two recurrent classes, R j = {1,2}, and 
R o = {5, 6, 7}. Assuming Xq = 3, find the probability that the chain gets absorbed in R ,. 



• Solution 

o Here, we can replace each recurrent class with one absorbing state. The resulting 
state diagram is shown in Figure 11.18 
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Figure 11.18- The state transition diagram in which we have 
replaced each recurrent class with one absorbing state. 

Now we can apply our standard methodology to find probability of absorption in 
state R j. In particular, define 

a i = /’(absorption in/lj \X Q = i ), for all i G S. 

By the above definition, we have a R =1, and a R = 0. To find the unknown 
values of a- s, we can use the following equations 


We obtain 


= T a kP,t ^riES. 


a 3 - 2 a R, + 2 a 4 


1 1 

— + ~ 
2 2 


+ 


1 1 1 

a 4 = 4% + 4 a 3 + 2 % 

1 1 

= ~ + -a,. 

4 4 3 

Solving the above equations, we obtain 



5 

Therefore, if X Q = 3, the chain will end up in class R j with probability a 3 = 
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Problem 3 

Consider the Markov chain of Example 2 . Again assume X Q = 3. We would like to find the 
expected time (number of steps) until the chain gets absorbed inf^ or R n . More specifically, 
let T be the absorption time, i.e., the first time the chain visits a state in R j or R^. We would 
like to find E\T\X Q = 3], 

• Solution 

o Here we follow our standard procedure for finding mean hitting times. Consider 
Figure 11.18. Let The the first time the chain visits R l orf? 0 .For all i G S', define 


t i = E[T\X Q = i\. 


By the above definition, we have t R ^ = t R = 0. To find and t 4 , we can use the 
following equations 

U = 1 + Z f kPit for 1 = 3 > 4 - 

k 


Specifically, we obtain 


h - 1 + 2% + i'4 


1 

2 ‘ 


1 + 


1 + + i'3 + 2'fl2 


1 + ~t v 
4 3 


Solving the above equations, we obtain 

12 10 

f 3 = y ’ l 4 = y ■ 

12 

Therefore, if A () = 3, it will take on average y steps until the chain gets absorbed 
in R\ or 
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Problem 4 

Consider the Markov chain shown in Figure 11.19. Assume X Q = 1, and let R be the first time 
that the chain returns to state 1, i.e., 

R = min {n > 1 :X n =1}. 


Find E[R \X Q = 1], 


1 

2 



1 

2 


Figure 11.19 - A state transition diagram. 

• Solution 

o In this question, we are asked to find the mean return time to state 1. Let r j be the 
mean return time to state 1, i.e., r \ = E\R \X () = 1], Then 

r \ = 1 + Z^l/b 

k 

where t k is the expected time until the chain hits state 1 given X () = k. 

Specifically, 

t x = 0 , 

t k =] + H t fkp for k ±\. 
j 

So, let's first find t k 's. We obtain 
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h ~ 1 + 3 f l + 3 r 3 


- 1 + J'3. 


1 1 

h = 1 + 2^3+ 2 f l 


- 1 + 2 '3- 


Solving the above equations, we obtain 


2, ?2 


7 

3' 


Now, we can write 


1 1 1 

r l = 1 + 4 f l + 2^2+ 4^3 

1 17 1 

= 1H— - OH —I— - 2 

4 2 3 4 


8 

3' 


Problem 5 

Consider the Markov chain shown in Figure 11.20. 


l 

4 
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Figure 11.20 - A state transition diagram. 

a. Is this chain irreducible? 

b. Is this chain aperiodic? 

c. Find the stationary distribution for this chain. 

d. Is the stationary distribution a limiting distribution for the chain? 

• Solution 

o a. The chain is irreducible since we can go from any state to any other states 
in a finite number of steps. 

b. The chain is aperiodic since there is a self-transition, i.e.,/> u > 0. 

c. To find the stationary distribution, we need to solve 


n 2 

*3 


1 1 1 

= 2 n l + 3^2 + ^3’ 

1 1 

= 4*1 + 2 7Z ' 3 ’ 

1 2 

= 4^1 + 3^2’ 


7T | 3“ 7 T ^ 3“ 7T 1. 


We find 


7Tj ~ 0.457, 7r 2 ~ 0.257, 7r 3 ~ 0.286 

d. The above stationary distribution is a limiting distribution for the chain 
because the chain is irreducible and aperiodic. 


Problem 6 


1 

Consider the Markov chain shown in Figure 11.21. Assume that ~ < p < 1. Does this chain 
have a limiting distribution? For all i,j E {0, 1, 2, •••}, find 

lim P(X n =j\X Q = i). 

ft—>00 


V V V 



1 —p 1 —p 1 —p 


Figure 11.21 - A state transition diagram. 

• Solution 
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o This chain is irreducible since all states communicate with each other. It is also 
aperiodic since it includes a self-transition, P 00 > 0. Let's write the equations for a 

stationary distribution. For state 0, we can write 

^0 = C 1 ~P) n o + C 1 ~P) n 1 ’ 


which results in 


7T, = 


1 ~p 


TCr 


For state 1, we can write 


ttj =pn 0 + (l ~p)n 2 

= (1 ~p)n x +(1 ~p)n 2 . 


which results in 


7T 9 = 


n i- 

1 ~p 


Similarly, for any j £ {1, 2, •••}, we obtain 


7Cj = an j _ l . 


where a = 


P 1 

—. Note that since - < p < 1, we conclude that a > 1. We obtain 

1 -p 2 ^ 


Kj = cPtcq, for j = 1,2, 


Finally, we must have 


i = Z 

7 = 0 


7T; 


= Yj°^ K 0 ’ 

7 = 0 


= gott 0 . 


(where a > 1) 


Therefore, the above equation cannot be satisfied if 7r 0 > 0. If 7T 0 = 0, then all jtj s 

must be zero, so they cannot sum to 1. We conclude that there is no stationary 
distribution. This means that either all states are transient, or all states are null 
recurrent. In either case, we have 


https://www.probabilitycourse.com/chapter11/11_2_7_solved_probs.php 


8/9 



10/8/2018 


Solved Problems 


lim P(X n =j \X Q = i) = 0, for all i,j. 

n —> go 


We will see how to figure out if the states are transient or null recurrent in the End 
of Chapter Problems (see Problem 15 in Section 11 .5 ). 


<— previous 
next —» 
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